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Verify Your Registration 



Enter the code shown 



More infe r? 
This helps arcc' i:" event automated registrations 
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Please review the following terms and indicate your agreement below Printable Version dx 
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Yahcc J Inc. ("Yahcc!") welcciaes ycu. Yahcc! 
prcvides its service tc ycu subject tc the 
following Terms cf Service ("T05"), which Fiay be • 

By clicking "I Agree" you agree ar>d consent to (a) the Yahoo 1 Terms of Service 
and Privacy Policy , and (b) receive required notices from Yahoo! electronically. 
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I Do Not Agree 



Verification technology developed in collaboration with Carnegie Mellon University 
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Verify Your Registration 

Enter the code shown. 



More info rP 
This helps arcc' prevent automated registrations. 




Terms of Service 



Please review the folio wing term* anC indicate your agreement below. Printable Vera ion £ 



1 . ACCEPTANCE OF TERMS 

Yahcc! Inc. ("Yahcc!") welcomes ycu. Yahcc J 
prcvides its service tc ycu subject to the 
fcllowing Terms cf Service ("TOS"), which may be 

Ey clicking "I Agree" you agree and con sent to (a i the Yahoo' Terms of Service 
and Privacy Policy , and (b) receive required notices from Yahoo! electronically. 
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Help! I've been waiting for 20 
minutes, and nothing happens 



CAPTCHA 

A program that can tell whether its 
user is a human or a computer 



[CACM] [EUROCRYPT] 



CAPTCHA 

A program that can generate and 
grade tests that: 

A. Most humans can pass 



B. Current computer 
programs cannot pass 



|P QUICKVOT 



Which is the best Computer Science 
Grad School in the US? 



Berkeley O 
CMU O 

Cornell O 



MIT O 

Washington O 
Stanford O 



vote 



(From www.slashdot.org) 



Applications 

Free e-mail services 

Data collection 

Worms 

Comment spam in blogs 

Preventing dictionary attacks 



Applications 

Free e-mail services 

Data collection 

Worms 

Comment spam in blogs 

Preventing dictionary attacks 



CAPTCHA Sweat Shops 

Spam companies hire humans to 
solve CAPTCHAs all day long 

$2.50 per hour for each human 
720 captchas per hour per human 



1/3 cent per account 
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Free Nude Photos 

Type the word in the box if you want 
to see the next picture 



Done 
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Roughly 60 million CAPTCHAs are 
solved every day 

10s of human time per CAPTCHA 



Can we use this human 
effort for good? 



Digitizing Books 
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Bell Henry, 
Belt Henry 
Bell Henry 
Bett Isaac, 
BeM Isabella 



CT BeH Henr 

OCR isn't perfect ^^effTTenrv 
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Scanned 
book 



Word that OCR 
can't read 



Randomly distorted 
Image of word 



We're Doing This For Realz! 



ARCHIVE 
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CAPTCHA 



reCAPTCHA comes with 
an audio test as well 



In the first month: 

Over 5,000 websites signed up 
Over 500,000 words digitized 



Protecting Email Addresses 

Posting your email address on the 

Web is dangerous! 



"biglou at cs dot emu dot edu" 



reCAPTCHA MailHide 
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Luis von 

Assistant P'cfe 
Computer s-ze 
Carnegie Mel to 



|bsg.,.@cs,cmu.i 
(reveal my en 



See my Goog 
Human Camp 



£ reCAPTCHA Stop Spam, Read Books - Windows Internet Explorer UEEL 
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Step spam ard help digitize books 1 
Cheoc out my new project: reCAPTCHA. 
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Enter the words in the picture to see this email address. 
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10,000 



Number of hours spent playing 

computer and video games of a typical 

21-year-old male in the United States 

(equivalent to 5 years of full time work!) 



Human Computation 



Labeling Images With Words 

Dick 
v Cheney 

Vice president 



Still an open problem 



Image Search On The Web 

Uses filenames and html text 



Microsoft Internet Explorer 



File Edit View Favorites Tools Help Jj|* 

Qeack - J 3 3 Vl 5Mrl:h Favorites (jf> Media ^3) (^ - 
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Web Images Groups News Froogle more » 
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_ J Preferences 



Strict SafeSearch is on 



Images Results 1 - 20 of about 1.129.000 for iloy [ definition ) with Safesejich «n (0.10 seconds) 

Show: All sizes- Large - Medium - Small 





free_top0403.gif dogs.jpg dogs.gif 

700 x 1129 pixels - 495 x 391 pixels - 45k 190 x 150 pixels - 7k 
210k sunsite.berkeley.edu/ sunsite.berkeley.edu/ 

www.nifty.com/vivadog/ T- T-Shirts/index.cgi? 

Shirts/dhland/dogs.jpg type=.. 

| More results from 
sunsite berkelev edu ] 



25304[1].jpg 

275 x 234 pixels - 6k 

www.dog-gone-pet- 

products.com/ 25304% 

5B1%5D.jpg 
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rabbit.jpg 

217x231 pixels -7k 

www.pet-clothing.net/ 

images/rabbit.jpg 



scottyphoto.jpg 

227 x 200 pixels - 6k 
www.pet-clothing.net/ 
images/scottyphoto.jpg 

[ More results from 
www.pet-clothinrj net | 



fundtag.gif 

325 x 230 pixels - 27k 
www.id-ideas.com/ 
g ifs/t a gs/fu ndtag.gif 



gotag5.gif 

325 x 214 pixels -31k 
www.id-ideas.com/ 
gifs/tags/gotag5.gif 
[ More results from 
www.id-ideas.com | 
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telescoping_pet_ramp- 
2.jpg 

483 x 475 pixels - 30k 
www.discountramps.com/ 
telescoping_pet_ramp-2.jpg 



blackt2.jpg blackt4.jpg waddling-d 

527 x 641 pixels - 50k 542 x 839 pixels - 79k 1024 ■ 768 p 

www.nhptv.org/natureworks/ www.nhptv.org/natureworks/ www.atpn 

graphicsZblackt2.jpg graphics/blackt4 jpg waddling-do 





shed.jpg curled.jpg 

640 x 480 pixels - 45k 570 x 372 pixels - 54k 

www.windmeupfarm.com/ www.steelypips.org/ 

shed.jpg images/dog/curled.jpg 



boxforweb.jpg 

1200 x 866 pixels- 514k 
www.doggonecrazy.ca/ 



Dogjpg 

1800 x 1179 pixels- i 
coastal.er.usgs.gov/ 
photos/I h/hike/Dog.j 
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HalfPmt1.JPG 

269 x 417 pixels ■ 
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Accessibility 

Majority of the web is not fully 
accessible to the visually impaired 

Reason: Most images don't have a 

caption 



Desiderata 

A method that can label 



all images on the web 
Fast and cheap 



Using Humans Cleverly 

The ESP game could label all images on 

google in a few weeks 



The ESP Game 

Two-player online game 

Partners don't know each other 
and can't communicate 

Object of the game: 
Type the same word 

Only thing in common is 
An image 



Q [CHI 2004] 



The ESP Game is Fun 



25 million agreements 
with 120,000 players 

There are many people that 
play over 20 hours a week 



Labeling The Entire Web 

5000 people playing simultaneously can 
label all images on Google in 2 months! 

Individual games in Yahoo! and MSN 
average over 5,000 players at a time 
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What About Cheating? 

We give players test images for which we 
know all the common labels: 

We only store a player's guesses if they 
successfully label the test images 



Search Results 
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Search Results 
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Sample Labels 




beach 

chairs 

sea 

people 

man 

woman 

plant 

ocean 

talking 

water 

porch 



Guesses Over Time 



britney 



crazy 
rehab 
shaved head 



Sample Labels 




ddam 
r. wilson 



Coming soon: 

Meet your soul mate 

through the game! 



Jumb 
yuck 



Why do people like the 

ESP Game? 



The ESP game gives its players a weird and 
beautiful sense of anonymous intimacy. 
On the one hand, you have no idea who 
your partner is. On the other hand, the 
two of you are bringing your minds 
together in a way that lovers would envy. 



Strangely addictive 

It's so much fun trying to guess 
what others think. You have to 
step outside of yourself to match 

It's fast-paced 
It's like crack!" 



Helps me learn English 



In other crowning achievements, I haven't 
played in at least a week, and I am still on 
the top ten list of players on the ESP game. 
Number eight if I remember correctly. 
God, my life is sad. 



" There are many fascinating things about the 
ESP Game: 

How it uses a game structure to build 
an index of images on the web 

What words become taboo from 
frequent use 

How interesting some of the images 
pulled randomly from the web are 

But by far the most intriguing aspect of the 
game is how often your random, unknown 
partner is a complete idiot. n 



Dear Luis von Ahn, 

Your game is a fraud 

I do research on ESP (extra sensory 
perception), and have an online ESP 
telepathy test on my website 

Please refrain from using the term 
ESP, as it makes us legitimate ESP 
researchers look like a fraud n 



The ESP Game is an 
"Algorithm" 

Input: an image 
Output: set of keywords 



Games With A Purpose 



Running a computation in people's 
brains instead of silicon processors 



Other Games With 
A Purpose 



Locating Objects in Images 



man 
woman 



ESP 
Game 



Locating Objects in Images 



i 




man 
woman 
water 
plant 



Such information would be extremely 
useful for computer vision research 
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eekoboom is Fun! 



27,000 players 
2,100,000 pieces of data 

Many played over 120 hours 
in the first 10 days! 
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1 


SWOOCIN 


3326313 


2 


TWO 


2222169 


3 


PEEKERBOOMER 


1856078 


4 
5 


DONNABEAR6100B 
SCRUFFY 


1651359 
1387500 


G 


BUTTHILDA 


1079438 


7 


DIBURIT 


1072137 


8 


SUBMOM4321 


1034602 


9 


COMFORT4MQM 


881404 


10 


DIANENOLAN • 


830056 




iDOS Carnegie Mellon Unij^r&ity, all ngnts I' tmW& j 



270 hours of game-play 



Player Comments 

Peekaboom gives the same feeling as 
combining gambling with charades 
while riding on a roller coaster. 
The good points are that you increase 
your intelligence, don't lose all your 
money and you don't fall off the ride. 
The bad point is that you look at your 
watch and eight hours have just 
disappeared! " 



One unfortunate side effect of playing 
so much in such a short time was a mild 
case of carpal tunnel syndrome in my 
right hand and forearm" 



Ronald 
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The Power Of Human Cycles 



CAPTCHA 



Problem 



Game 



www.recaptcha.net 



www.espgame.org 
www.peekaboom.org 



Peekaboom Crew 

Luis von Ahn 

Roy Liu 

reCAPTCHA Crew 

Luis von Ahn 

Ben Maurer 

Mike Crawford 



Speaker's Entertainment 

Provided By 

Homestarrunner.Com 
uoogie, mc 

The MacArthur Foundation 

David Stork 

Latanya Sweeney 



